The conventional methods for merging video images include the "blue screen" or "chromo-key" techniques. For example, these techniques are used in the broadcast industry to show weather forecasters positioned in front of weather maps. In this technology, the forecaster is filmed in front of a screen having a particular pre-determined color. The screen color is usually blue, hence, the name of this technique. The technique calls for substituting the second image (the weather map) at all points in the first image (the forecaster) where the signal contains the blue color. The result is that the image of the map and forecaster will appear to merge. However, this method has two short comings. First, the forecaster must not have any item on his person that is the same color as the blue color of the screen. If he does, and the second image is merged with the first image, any item on the forecaster having the predetermined blue color will appear as a hole. Second, this technique does not provide for the integration of the two images. That is, the two images are merely superimposed. That is why the forecaster must make sweeping and general motions to indicate positions on the map. This limitation is important when considering this technique for an advertising application.
There are of course other techniques which enable the insertion of video images into an existing image which are basically employed at the studio and which utilize complicated console devices to superimpose a video picture on another video picture. Essentially the present technique involves a completely different approach with great applicability to television advertising. As will be explained, the present application relates to a method of advertising using existing television transmission facilities in which the advertiser selects predetermined areas in a video scene. These areas are then recognized using existing technology as pattern recognition techniques and the content of these areas is replaced by inserting an image or images of the advertisers choosing. In order to accomplish the present invention one must recognize the selected predetermined area and replace the same with advertising indicia in real time. The inserted indicia is then blended into the original video image in such a way that it appears part of the television scene and cannot be otherwise discerned by a typical viewer. The inserted advertising or messages can further be highlighted to the viewer by moving the message, changing color, associating the message with sound and otherwise imparting modulation to the message to distinguish the message during the course of the television program. Such techniques will be further described. Basically, it is believed that the techniques to be described herein have extreme usefulness and applicability for the display of advertising in television sporting events such as tennis matches, boxing, baseball, football and so on. While there is particular applicability to such events the techniques described may be adapted to all formats of video presentation and have general applicability to the field of advertising.
As will be described and in order to accomplish this result, one must basically select an object or an area within a given video image. For example the area selected may be a televised tennis court. This area is then recognized in each of a subsequent stream of video images and once recognized the video content of that image is replaced with a desired content which may be an advertising commercial, logo or some other matter. The insertion and replacement of the existing video image with the new material is accomplished completely independent of the size of the image in subsequent television scenes or frames. The system described herein will always recognize the particular area selected no matter what the size of the area is with respect to the remaining television picture. This, therefore, creates the illusion to the viewer that the replaced subject matter is actually that subject matter which is being televised in real time. Thus, the methods and apparatus described enable advertisers to add or merge images, messages or slogans to preexisting video images in such a way that they appear to be part of the original image even when the original video stream is of a live event being distributed in real time.
Thus, the techniques described herein are superior to prior art techniques of inserting video information. In order to do and accomplish the results of this particular application an electronic device is required to select, recognize and substitute images to be broadcast on live televised events. The device detects part or all of an object or objects within a video image and uses the position of the object or part thereof as a reference. The device then positions another still or video image into the original image at the reference location. The still or video image is positioned accurately so that the final composite image appears as though it is part of the original scene. That is, the added image is seamlessly and realistically incorporated into the original event. The appearance of the added image will conform to the appearance of the original scene when the scene is moved, panned, magnified, zoomed or otherwise altered in size or perspective. The System can also compensate for distortions by mapping the distortions in a televised scene and using the mapped distortions in future processing. All this must be done in real time, or with a processing delay being sufficiently small that it can be applied to events being transmitted live. The "Burt Pyramid" is the most appropriate way to do this amount of computation on a video image in such a short time with existing technology. This is a well known method of processing, analyzing and/or synthesizing signals and Real-Time Hierarchy Pyramid Signal Processing Apparatus by C. R. Carlson, et al. and issued on Jun. 16, 1987. The Burt Pyramid is also discussed in the publications entitled "Fast Algorithms for Estimating Local Image Properties", by Peter J. Burt, Computer Vision, Graphics and Image Processing 21, pp. 368-382, 1983 and "Pyramid-Based Extraction of Local Image Features with Application to Motion and Texture Analysis", by Peter J. Burt, SPIE, Vol. 360, pp. 114-124.
The "chromo-key" or "blue-screen" technology may be used in certain applications as an adjunct to the pattern recognition method. This technology is one way of distinguishing non-replaceable, foreground pixels from replaceable, background pixels. For instance, in a sport such as tennis, it is usually necessary to ensure that only the tennis court is replaced with a message merged onto the field of play. The players or their equipment should not be affected. One way to accomplish this is to replace only the green colored pixels that are in the merged image and are within the color range of the court. Alternate methods of distinguishing between replaceable and non-replaceable pixels include storing an image of the unoccupied court and subsequently using that image as a comparison to determine which is the player/equipment and which is the background. Another method uses differential motion between the objects or markings on the court and the equipment/players moving over the court to distinguish between replaceable and non-replaceable pixels.
Although the pattern recognition software using the "Burt Pyramid" algorithms is fast, it still takes a finite time to operate. Therefore, it may be necessary to introduce a time delay into the system. The delay may be one or two frames, or it may be longer. A frame store or other similar means can be used to temporarily locate a small number of the video screens or frames. Incorporating a time delay unit into the present invention will be discussed hereinafter.
Other conventional methods of processing video signals and images or the like are presented below. Some of the disclosed methods also use the Burt Pyramid.
An example of an apparatus that analyzes signals and the frequency spectrum for the frequency components of the signals is disclosed in U.S. Pat. No. 3,731,188, Signal Analysis of Multiplicatively Related Frequency Components in a Complex Signal, which issued May 1, 1973 to S. E. Smith. Other patents disclosing subject matter related thereto are U.S. Pat. Nos. 3,140,710, 3,315,222, 3,470,468, 3,473,121 and 3,582,957.
An example of an image processing method for reducing noise in the image is disclosed in U.S. Pat. No. 4,442,454, Image Processing Method Using a Block Overlap Transformation Procedure, which issued Apr. 10, 1984 to P. G. Powell. In the method disclosed, the image is divided into blocks of image elements which are transformed according to the linear transform procedure. Other patents disclosing subject matter related thereto are U.S. Pat. Nos. 3,715,477, 3,996,421, 4,055,756, 4,057,828, 4,134,134, 4,205,341, 4,242,705 and 4,245,248.
An example of a method and apparatus for transforming numerical signal data of a video signal is disclosed in U.S. Pat. No. 4,447,886, Triangle and Pyramid Signal Transforms and Apparatus, which issued May 8, 1984 to G. W. Meeker. As disclosed in this patent, the numerical signal data of the video signal is subjected to a transform domain where the signal is processed and reconstructed into the original domain to provide a replica of the original data. Other patents disclosing subject matter related thereto are U.S. Pat. Nos. 2,921,124, 3,051,778, 3,742,201, 4,222,076, 4,261,018, 4,261,043 and 4,286,330.
An example of an image-processing system employing coring techniques to reduce a noise component of an image signal is disclosed in U.S. Pat. No. 4,523,230, System for Coring an Image-Representing Signal, which issued Jun. 11, 1985 to C. R. Carlson, et al. The Burt Pyramid spectrum analyzer and synthesizer (U.S. Pat. No. 4,674,125) may be employed with the system disclosed in the '230 patent, so that the system can operate in real time on a video signal of a scanned television image. Other patents disclosing subject matter related thereto are U.S. Pat. Nos. 4,163,258 and 4,463,381.
An example of a method and an apparatus for reducing image data while retaining the ability to observe objects with high spatial resolution is disclosed in U.S. Pat. No. 4,692,806, Image-Data Reduction Technique, which issued Sep. 8, 1987 to C. R. Carlson, et al. Other patents disclosing subject matter related thereto are U.S. Pat. Nos. 3,016,518, 3,983,328, 4,405,943 and 4,513,317.
An example of an image processing method for filling in one or more void defects in an image to provide a natural looking processed image is disclosed in U.S. Pat. No. 4,698,843, Method for Compensating for Void-Defects in Images, which issued Oct. 6, 1987 to P. J. Burt, et al. This invention discloses employing The Burt Pyramid as disclosed in U.S. Pat. No. 4,674,125. Other patents disclosing subject matter related thereto are U.S. Pat. Nos. 3,737,855, 3,887,762, 3,973,239, 4,000,399, 4,010,446 and 4,661,986.
Thus as one will ascertain the advertising method and apparatus of this invention requires very precise positioning of a new image into an existing image. The techniques to be described employ pattern recognition apparatus which recognizes pre-selected features or portions of a typical television scene. These features can then be used to locate the position, size and perspective of an artificial electronic advertising message which is added in the exact congruency with the replaced recognized image and appears to the end user as if it were part of the original scene. The added material is inserted into the pre-selected scene independent of size of the selected feature as varying on a frame to frame basis.